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Abstract 


This study was aimed to investigate effect of National Examination (UN) and curriculum also their interaction 
effect on students’ performance in solving TIMSS model Mathematics problems. This study was carried out 
with 300 8" graders of East Kalimantan students. The participants were from three mathematic National 
Examination result school categories on 2016 (low, moderate and high) and two implemented curriculum types 
(KTSP and Currciculum 2013). Data collection used 28 items TIMSS model Mathematics problems (a = 0.837) 
which contained content and cognitive domain. Data analysis used two ways analysis of variance. Data analysis 
revealed that there were significance main effect of national examination and curriculums, also interaction 
effect between of them on mathematic achievement. There were significance main effect of national 
examination and curriculums on content domain. But, there was no significance interaction effect between of 
them on performance in solving geometry. There were significance main effect of national examination and 
curriculums on cognitive domain. But, there was no significance interaction effect between of them on knowing 
thinking level. 
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INTRODUCTION 

Globalization and science technology development in 21“ century strived international 
citizen for mastering some knowledges and skills, one of them is mathematics. National 
Council of Teacher of Mathematics (2000) explained that, In this changing world, those who 
understand and can do mathematics will have significantly enhanced opportunities and 
options for shaping their futures. Mathematical competence opens doors to productive 
futures. A lack of mathematical competence keeps those doors closed. In order to study and 
canal information about citizen readiness to overcome the challenges, there are many 
international studies about mathematics performance. One of them which participated by 
Indonesia is Trends in International Mathematics and Science (TIMSS). 

TIMSS is a study which measures achievement of mathematics and science for 4™ and 
8" grade (Mullis, Martin, Ruddock, O’Sullivan and Preuschoff, 2009). The purpose of 
TIMSS is to inform about educational policies of participant countries. Yilmaz and Hanci 
(2015) stated that TIMSS study collects comparative data about national education systems 


Prisma Sains: Jurnal Pengkajian Ilmu dan Pembelajaran Matematika dan IPA IKIP Mataram, June 2020. Vol. 8, No.1 | 1 


Hamidy et al The effect of national examination and ........... 


so they contribute to improvement of education, especially in mathematics and science. The 
statement is parallel to Martin and Mullis (2006) that TIMSS was designed to collect 
information that could be used to improve mathematics and science learning in the world. By 
collecting data periodically, TIMSS present trend of education. The monitoring extends 
important information to develop educational policies, improve people accountability and 
identify increasing or decreasing of achievement. 

Term of “achievement” relate to students’ achievement in mastering several abilities in 
a subject which has formulated in curriculum (Mullis, et. al, 2009). Assessment of the 
achievement use problems which have developed based on the framework. The assessment 
framework was divided into two domain, conten and cognitive (Mullis et al, 2009). Content 
domain is collection of mathematical content which assessed by TIMSS, they are number, 
algebra, geometry and data & chance. Cognitive domain concern to skills in thinking process, 
from lower order to higher order (Budiman & Jailani, 2014), which include knowing, 
applying and reasoning. 

Mathematics performance of Indonesian student are not only assessed by international 
assessment, but also national examination. National examination is national assessment that 
measure compentencies achievement in several subjects and referred to graduate competency 
standards (re: Standar Kompetensi Lulusan, SKL) (Badan Standar Nasional Pendidikan, 
2017). SKL are parts of educational curriculum of Indonesia. Thus, based on frameworks 
problems, both national examination and TIMSS referred to educational curriculum. It means 
that mathematics performance which assessed by national examination related to TIMSS, 
although both of them have different type of problems (Stacey, 2011; Carnoy, et al., 2016). 

Since 2014, TIMSS model Mathematics problems started embedded to national 
examination in order to improve mathematic achievement internationally. But, the result of 
mathematics national examination was not improved, year by year, even it was declined 
(Retnawati, Kartowagiran, Arlinwibowo and Sulistyaningsih, 2017). It was parallel to 
Indonesian mathematics achievement in TIMSS, which was still low and has not get 
significant improvement (Balitbang Kemendikbud, 2011; Mullis, Martin, Foy, and Arora, 
2012). Thus, it assumed that there is a relationship between students’ performance in solving 
mathematics national examination and mathematics achievement in TIMSS. 

Still, mathematics achievement also affected by another factors, which come from 
school or student’ characteristics. Several studies (ex. Carnoy, et.al, 2016; Lam and Lau, 
2014; McConney and Perry, 2010; Jerrim, Lopez-Agudo, and Marcenaro-Gutierrez, 2020; 
Yun, 2019) imply that several factors which relate characteristics and quality of school and 
student contribute to students’ performance. Study of Carnoy, et.al (2016) found that learning 
opportunity and teaching quality improvement generate vary effect to mathematics 
achievement, it depends to parents education level and mathematics school performance. One 
of Lam and Lau (2014) findings is number of student in a class and socio-economic status 
was strong predictor of PISA score. On the contrary, Ghasemi and Burley (2019) found that 
socio-cultural, politics, equity have not affected to mathematics performance. 

NCTM (2000) stated that main factor determined mathematics performance is applied 
curriculum. Scheerens (2002) also claimed that education effectivity which related to students 
achievement can be explained by three main factors, one of them is learning effectivity which 
related to curriculum. Curriculum consists of three aspects, there are the intended curriculum, 
the implemented curriculum and the achieved curriculum (Martin and Mullis, 2006). Thus, 
mathematics performance affected by planning, implementing and assessing through 
mathematics competencies which expected to be mastered by students. 

At Indonesia, there are still different curriculum which implemented by schools, KTSP 
(School Based Curriculum) and Kurikulum 2013 (K13). Besides the emphasis on knowledge 
and skills, the difference between KTSP and K13 also lies in the learning approach and 
assessment system (Retnawati, Hadi, and Nugraha, 2016). K13 emphasizes student-centered 
learning rather than teacher-centered learning. In assessing students’ performance, KTSP use 
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tests that emphasize more aspects of knowledge and skills. Meanwhile, K13 use authentic 
assessments that measure students' attitudes, knowledge, and skills based on the learning 
process and outcomes. In addition, K13 learning promotes Higher Order Thinking Skills 
(HOTS) more than KTSP (Suryapuspitarini, Wardono, & Kartono, 2018). 

Indonesian curriculum changing from KTSP to K13 was based on the low ranking of 
Indonesian students on PISA and TIMSS (Kemendikbud, 2012). That is, the implementation 
of the K13 aimed to improve students' mathematical performance internationally. Thus, the 
implementation of K13 should affect a positive influence on students’ mathematics 
performance than KTSP. However, yet found studies that evaluating K13 in improving the 
ranking of Indonesian students in PISA and TIMSS. 

Therefore, it is necessary to investigate the effect of the National Examination and 
curriculum on students’ performance in solving TIMSS model Mathematics problems. This 
study was aimed to investigate effect of National Examination (UN) and curriculum also their 
interaction effect on students’ mathematics achievements, that is students performance in 
solving TIMSS model Mathematics problems, both in general and based on domain. This 
study is different from previous TIMSS studies in Indonesia which still only describe 
students performance in solving TIMSS problems, but have not investigated the factors that 
affect it specifically. 


METHOD 

This study was a survey research. This study was conducted to 12 Junior High Schools 
distributed over East Kalimantan. This study was carried out with 300 8™ graders of East 
Kalimantan students. The participants were from three mathematic National Examination 
result school categories on 2016 (low, moderate and high) and two implemented curriculum 
types (KTSP and K13). Gender based distribution was fairly balanced, that is 151 boys and 
149 girls. 

Data collection used 28 items TIMSS model Mathematics task. The task was design 
based on TIMSS framework which contained content and cognitive domain. Content domain 
consist of number, algebra, geometry and data & chance. Cognitive domain consist of 
knowing, applying and reasoning. The task has been validated by expert judgement and 
restricted trial. After validation, the task has been revised based on expert suggestions. The 
trial found that the reliability estimation was 0.837. The measure meet the minimum standard 
reliability instrument to measure group of individuals (Ebel & Frisbie, 1991). 

The obtained data was score test then converted to percentage of correct answer, both 
the TIMSS score in general and based on cognitive and content domain. The data were 
described by average percentage of correct answer based on National Examination and 
curriculum groups. Furthermore, the data were analyzed using two-way analysis of variance 
to investigate: 1) the effect of the National Examination, curriculum and their interactions on 
mathematics achievement in general; and 2) the effect of the National Examination, 
curriculum and their interaction on mathematics achievement based on content and cognitive 
domains. Post hoc test was conducted if the main effect was significant, both in general and 
based on cognitive and content domain. 


RESULTS AND DISCUSSION 
Results 

Generally, the average percentage of correct answers to the TIMSS model Mathematics 
problems is 27.76 with a standard deviation of 19.85. This percentage is relatively low 
because it is below to the 2015 international TIMSS average (Mullis, Martin, Foy, and 
Hooper, 2016). The highest percentage is 96.77, while the lowest is 3.23. The average 
percentage of correct answers to the TIMSS model Mathematics problems based on the 
National Examination results category and the implemented curriculum is shown in Table 1. 
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Table 1. The average percentage of correct answers to the TIMSS model Mathematics 


problems 
KTSP K13 
High 38,34 48,82 
Moderate 14,85 19,22 
Low 15,86 31,68 


Based on the results of the analysis of variance (Table 2), the National Examination, 
curriculum and interaction both of them have a significant effect on the percentage of correct 
answers to the TIMSS model Mathematics problems in general. Based on Table 1 and Table 
2, the percentage of correct answers to the TIMSS model Mathematics problems done by 
students from schools that have implemented K13 is significantly higher than students from 
schools that have implemented KTSP. Post hoc test results of the National Examination effect 
on the percentage of correct answers to the TIMSS model Mathematics problems are shown 
in Table 3. 

Table 2. Anova of the percentage of correct answers to the TIMSS model Mathematics 


problems 
Factors Mean Square F Sig 
Curriculum 6638.96 32.99 0.000 
National Examination 15947.2 79.25 0.000 
Curriculum * National Examination 737.69 3.67 0.027 


Table 3. Post hoc test of the National Examination (NE) effect on the percentage of correct 
answers to the TIMSS model Mathematics problems 


NE Category Average Diff. Sig. 

High — Moderate 29.19 0.000 
High — Low 25.81 0.000 
Moderate — Low -3.39 0277 


Table 3 shows that the percentage of correct answers to the TIMSS model Mathematics 
problems done by students from high school category was significantly higher than students 
from moderate and low school category. While the percentage of correct answers to the 
TIMSS model Mathematics problems between students from the moderate and low category 
schools did not differ significantly, although the students percentage of correct answers from 
the low schools was higher than moderate schools. 

In terms of the content domain, the average percentage of correct answers to the 
TIMSS model Mathematics problems is shown in Figure 1. It shows that the average 
percentage of correct answers to the TIMSS model Mathematics problems for each content 
domain is below to the 2015 TIMSS international average. 
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==@— The average percentage of correct answers 


==@=—= 2015 TIMSS average 


Figure 1. The average percentage of correct answers to the TIMSS model Mathematics 
problems for each content domain 
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The average percentage of correct answers to the TIMSS model Mathematics problems 
for each content domain, the National Examination results category and the implemented 
curriculum also the analysis of variance are shown in Table 4 and Table 5. Based on the 
results of analysis of variance (Table 5), the National Examination and curriculum have a 
significant effect on the performance in solving each content domain, but the interaction of 
them did not significantly affect to the performance in solving geometry problems. Based on 
Table 4 and Table 5, the percentage of correct answers to the TIMSS model Mathematics 
problems done by students from schools which implemented K13 was significantly higher 
than students from schools which implemented KTSP for each content. 

Table 4. The average percentage of correct answers to the TIMSS model Mathematics 
problems for each content domain 


KTSP K13 
Number High 58.39 62.04 
Moderate 15.09 23.40 
Low 16.29 43.18 
Algebra High 23.08 39.02 
Moderate 11.56 13.56 
Low 8.85 14.73 
Geometry High 33:97 47.97 
Moderate 18.24 23:76 
Low 23.38 33.92 
Data & Chance High 26.28 38.51 
Moderate 15.40 14.54 
Low 16.89 30.95 


Table 5. Anova of the percentage of correct answers to the TIMSS model Mathematics 
problems for each content domain 


Factors Content Domain Mean F Sig 
Square 

Curriculum Number 10650 26.80 0.000 
Algebra 4006.4 15.98 0.000 
Geometry 6378.1 16.26 0.000 
Data and Chance 4557.7. 11.78 0.000 

National Examination Number 37942 95.48 0.000 
Algebra 9576.5 38.20 0.000 
Geometry 8692.5 22.16 0.000 
Data and Chance 6642.0 17.17 0.000 

Curriculum * National Examination Number 3059.5 7.69 0.001 
Algebra 1087.7 4.34 0.014 
Geometry 402.6 1.03 0.360 
Data and Chance 1518.8 3.92 0.021 


Post hoc test results of the National Examination effect on the percentage of correct 
answers to the TIMSS model Mathematics problems for each content domain are shown in 
Table 6. It shows that the percentage of correct answers to the TIMSS model Mathematics 
problems done by students from high school category was significantly higher than students 
from moderate and low school category for each content domain. Meanwhile, the percentage 
of correct answers to the TIMSS model Mathematics problems between students from the 
moderate and low category schools did not differ significantly for each content domain, 
although the students percentage of correct answers from the low schools was higher than 
moderate schools. 
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In terms of the cognitive domain, the average percentage of correct answers to the 
TIMSS model Mathematics problems is shown in Figure 2. It shows that the average 
percentage of correct answers to the TIMSS model Mathematics problems for each cognitive 
domain is below to the 2015 TIMSS international average. 

The average percentage of correct answers to the TIMSS model Mathematics problems 
for each cognitive domain, the National Examination results category and the implemented 
curriculum also the analysis of variance are shown in Table 7 and Table 8. Based on the 
results of analysis of variance (Table 8), the National Examination and curriculum have a 
significant effect on the performance in solving each cognitive domain, but the interaction of 
them did not significantly affect to the performance in solving knowing problems. Based on 
Table 7 and Table 8, the percentage of correct answers to the TIMSS model Mathematics 
problems done by students from schools which implemented K13 was significantly higher 
than students from schools which implemented KTSP for each cognitive domain. 

Table 6. Post hoc test of the National Examination (NE) effect on the percentage of correct 
answers to the TIMSS model Mathematics problems for each content domain 


Content Domain NE Category Mean Diff. Sig. 
Number High—Moderate 42.09 0.000 
High-Low Shiai 0.000 
Moderate—Low -4.82 0.203 
Algebra High—Moderate 22.38 0.000 
High-Low 24.38 0.000 
Moderate—Low 2.00 0.645 
Geometry High—Moderate 23.49 0.000 
High-Low 17.99 0.000 
Moderate—Low -5.49 0.123 
Data and Chance High—Moderate 20.33 0.000 
High-Low 14.50 0.000 
Moderate—Low -5.83 0.111 
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Figure 2. The average percentage of correct answers to the TIMSS model Mathematics 
problems for each cognitive domain 


Post hoc test results of the National Examination effect on the percentage of correct 
answers to the TIMSS model Mathematics problems for each cognitive domain are shown in 
Table 9. It shows that the percentage of correct answers to the TIMSS model Mathematics 
problems done by students from high school category was significantly higher than students 
from moderate and low school category for each cognitive domain. Meanwhile, the 
percentage of correct answers to the TIMSS model Mathematics problems between students 
from the moderate and low category schools did not differ significantly for each cognitive 
domain, although the students percentage of correct answers from the low schools was higher 
than moderate schools. 
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Table 7. The average percentage of correct answers to the TIMSS model Mathematics 
problems for each cognitive domain 


KTSP Kurikulum 2013 
Knowing High 39.86 54.05 
Moderate 18.87 25.72 
Low 23.48 37.01 
Applying High 37.82 48.08 
Moderate 14.93 16.84 
Low 13.19 29.46 
Reasoning High a7 02 42.74 
Moderate 9.19 13.83 
Low 9.37 27.68 


Table 8. Anova of the percentage of correct answers to the TIMSS model Mathematics 
problems for each cognitive domain 


Factors Cognitive Mean Square F Sig 
Domain 

Curriculum Knowing 8438.2 29.85 0.000 
Applying 5707.6 20.21 0.000 
Reasoning 5792.8 18.99 0.000 

National Examination Knowing 13420 47.47 0.000 
Applying 17018 60.26 0.000 
Reasoning 18258 59.86 0.000 

Curriculum * National Examination Knowing 375.8 1.33 0.266 
Applying 1174.6 4.16 0.017 
Reasoning 1211.9 397 0.020 


Table 9. Post hoc test of the National Examination (NE) effect on the percentage of correct 
answers to the TIMSS model Mathematics problems for each cognitive domain 


Cognitive Domain NE Category Mean Diff. Sig. 
Knowing High—Moderate 28.2/ 0.000 
High-Low 23.09 0.000 
Moderate—Low -5.18 0.076 
Applying High—Moderate 29.58 0.000 
High—Low 27.67 0.000 
Moderate—Low -1.92 0.699 
Reasoning High—Moderate 29.87 0.000 
High—Low 26.75 0.000 
Moderate—Low -3.12 0.620 
Discussion 


The overall TIMSS score shows that the mathematics achievement of East 
Kalimantan junior high school students is relatively low. Even so, the score was higher than 
the TIMSS score of Indonesian junior high school students in TIMSS 2011 (Mullis, Martin, 
Foy, and Aurora, 2012). In term of content and cognitive domains, students' mathematics 
achievements are still below the international average. Students’ mathematics achievement 
which are below to the international average shows that most students have not yet reached 
the mathematical competencies formulated in the international curriculum. 

The low mathematics achievement could be caused by various factors, include those 
from schools, students and environment. These various factors can affect mathematics 
achievement in general and specifically to the content and cognitive domains. The findings 
showed that the National Examination had a significant effect on students' mathematics 
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achievements both in general and in terms of the content and cognitive domains. That is, the 
performance in solving the National Examination problems is an important factor that affect 
students’ success in solving TIMSS problems. This is caused by the suitability of the 
mathematics problems characteristics in the National Examination and TIMSS, both in terms 
of the scope of the content and the level of thinking (Stacey, 2011; Carnoy, et.al, 2016). 

Based on the post hoc tests, it was found that student success in the National 
Examination had a positive effect on students' mathematics achievement. It means that 
students from schools with a high National Examination category achieved higher 
mathematics achievements than students from the low and moderate school categories. So, if 
the higher the National Examination score in mathematics then the higher student's 
mathematics achievement in TIMSS. This is in line with the findings of Yilmaz and Hanci's 
(2015), that the higher the mathematics scores of students in school, the higher the TIMSS 
score that students can achieve. Thus, embedding TIMSS type problems into National 
Examination test is an appropriate effort to learn students in improving their mathematical 
abilities internationally (Retnawati, Kartowagiran, Arlinwibowo and Sulistyaningsih, 2017). 

However, a significant positive effect was not shown between students from the 
moderate and low National Examination school categories. In each content and cognitive 
domain, the students performance from low National Examination school categories 
consistently higher than students from moderate National Examination schools, although it is 
not significant. It showed that there is a large performance gap between students from schools 
with a high National Examination category and students from schools with moderate and low 
National Examination categories. 

Furthermore, the findings showed that the curriculum implemented by the school 
affected students' mathematics achievements in general and specifically in the content and 
cognitive domains. That is, the curriculum is also an important factor that affects students' 
mathematics achievement in TIMSS. This is in line with the purpose of TIMSS, which is to 
measure student achievement in mastering various abilities that have been formulated in the 
education curriculum (Mullis, et.al., 2009). Thus, mathematics achievement reflects the 
effectiveness of the curriculum applied (NCTM, 2000; Scheerens, 2002). The effectiveness of 
the curriculum includes the range of content and competencies to be achieved, the method of 
its implementation, as well as the form of evaluation and assessment of the competency 
outcomes that have been formulated. 

Specifically, the findings also showed that the students mathematics achievements 
from schools which implemented K13 were significantly higher than students from KTSP 
one, both in general and based on content and cognitive domains. This confirms the 
assumption that have been proposed previously, that aspects, approaches and assessment 
methods developed in K13 are oriented towards students' preparation in facing global 
challenges that appropriatewith the objectives of TIMSS (Retnawati, Hadi, and Nugraha, 
2016; 2013 Curriculum Documents; Mullis, et.al., 2009). So, the scope of the content and the 
development of thinking skills in TIMSS are more accommodated in K13 than KTSP. 

Although it is more in line with the TIMSS curriculum, K13 policy has not succeeded 
in improving Indonesian students mathematics achievements at TIMSS to come out from the 
bottom rank internationally. This is due to the implementation of K13 which is not yet 
appropriate and optimal (Jaedun, Hariyanto, and Rahardjo, 2014; Trisnawati, Gunawan, and 
Nongkeng, 2017), especially HOTS learning. Retnawati, Dyjidu, Kartianom, Apino, and 
Anazifa (2018) found that teachers' understanding and ability to teach, assess and solve 
HOTS problems was still low. HOTS questions have also not been introduced to students 
evenly (Witri, Putra, and Gustina, 2014). In addition, efforts to improve students' 
mathematical achievements internationally are not enough just to reform the education 
curriculum. Saoudi, Chroqui, and Okar, (2019) explained that other efforts needed to improve 
student achievement internationally, including the recruitment and training of teachers in 
order to achieve minimum standards of qualification, involving various stakeholders in 
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formulating educational programs that fit the needs, school systems that reflect the real 
world, focus on aspects of learning and build a culture of evaluation and accountability. 

The interaction between the factors of the National Examination and the implemented 
curriculum also has a significant effect on students’ mathematics achievement in general. This 
significant interaction is similar to the findings by Indartono and Hamidy (2019) that 
curriculum differences have a moderating effect on the effect of National Examination results 
on international test results such as PISA and TIMSS. This is due to the National 
Examination which is still related to the curriculum, which is part of a centralized assessment 
by the government that refers to certain graduation standards (BSNP, 2017). Thus, the 
differences in the implemented curriculum by schools in Indonesia also have an impact on the 
results of the National Examination that they obtained, considering that the National 
Examination is still enforced equally to schools that implement either KTSP or K13. 

In terms of content domain, the interaction between the factors of the National 
Examination and curriculum has no significant effect on the content of the geometry. It 
occurred because there are no differences in the scope of geometry content between those 
formulated in the curriculum and those tested on the National Examination (Permendiknas 
No. 22/2006; Permendikbud RI No. 24/2016). Whereas in other content domains, there is a 
change in content coverage both in K13 compared to KTSP and in the National Examination 
problems. 

In terms of cognitive domain, the interaction between the factors of the National 
Examination and curriculum has no significant effect on the level of thinking knowing. It 
shows that the ability to think low level has been accommodated in the curriculum and the 
National Examination. While the abilities to think at a higher level are more emphasized on 
K13 than KTSP. In addition, the ability to solve problems with a higher level of thinking is 
the key to students’ success in achieving high scores in National Examination and TIMSS. 


CONCLUSION 

Based on the findings and discussion, it concluded that: 1) National Examination and 
curriculum as well as the interaction of both of them significantly affect students’ 
mathematics achievement; 2) National Examination and curriculum have a significant effect 
on the ability for each content domain, but the interaction between them did not significantly 
affect to the ability to solve geometry problems; 3) National Examination and curriculum 
have a significant effect on the ability for each cognitive domain, but the interaction of them 
did not significantly affect to the ability to know (knowing). 


RECOMMENDATION 

The results of the study have implications for improving the quality of the National 
Examination and education curriculum in Indonesia in order to improve the mathematics 
achievement of Indonesian students at the international level. On the one hand, the sample 
used in this study is different from the sample used by TIMSS to assess the mathematics 
achievements of Indonesian students. So that the comparison of performance which presented 
between East Kalimantan and Indonesian students is not entirely equivalent. Therefore, it is 
necessary to study the data sources and sampling techniques used in TIMSS for next 
subsequent research. 
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